How to be a CPAN tester

Barbie

Who Am I?

• Barbie
  • CPAN Tester (2003-2004)
  • Co-developer of CPAN-YACSmoke
  • Maintainer of CPAN Testers Statistics
  • Maintainer of CPAN Testers Wiki
  • Maintainer of CPAN Testers Reports
  • Caretaker for CPAN Testers

MANIFEST

• What is CPAN Testing?
• How CPAN Testing Works
• The CPAN Testers Community
• The Future
• Questions

What is CPAN testing?

• Tests CPAN distributions
• Many distributed testers
• Multiple platforms
• Multiple perls
• Structured feedback mechanism
• Anarchic (Chaotic?)

Who benefits from CPAN testing?

• Authors

Who benefits from CPAN testing?

• Authors
• Users
  • Corporates
  • Small Business
  • Individuals

Who benefits from CPAN testing?

• Authors
• Users
  • Corporates
  • Small Business
  • Individuals
• Perl

How does CPAN testing work?

• PAUSE announces a new upload

How does CPAN testing work?

• PAUSE announces a new upload
• Distribution downloaded

How does CPAN testing work?

• PAUSE announces a new upload
• Distribution downloaded
• Testers run tests (manual or automated)

How does CPAN testing work?

• PAUSE announces a new upload
• Distribution downloaded
• Testers run tests (manual or automated)
• Tests results sent to CPAN Testers mailing list
  • http://lists.cpan.org/showlist.cgi?name=cpan-testers
  • nntp://nntp.perl.org/perl.cpan.testers

How does CPAN testing work?

• PAUSE announces a new upload
• Distribution downloaded
• Testers run tests (manual or automated)
• Tests results sent to CPAN Testers mailing list
  • http://lists.cpan.org/showlist.cgi?name=cpan-testers
  • nntp://nntp.perl.org/perl.cpan.testers
• Results tabulated on CPAN Testers Reports

How does CPAN testing work?

• PAUSE announces a new upload
• Distribution downloaded
• Testers run tests (manual or automated)
• Tests results sent to CPAN Testers mailing list
  • http://lists.cpan.org/showlist.cgi?name=cpan-testers
  • nntp://nntp.perl.org/perl.cpan.testers
• Results tabulated on CPAN Testers Reports
• Author notified of FAIL and UNKNOWN reports

Types of Reports

• PASS - all the tests in test suite passed
• NA - distribution not installable
• FAIL - one or more tests failed
• UNKNOWN - distribution has no test :(

What to test?

• Correctly formatted distribution uploads

My-Module-0.01.tar.gz
My-Module-0.01.tar.bz2
My-Module-0.01.tgz
My-Module-0.01.zip

My-Module.tar.gz
My-Module-0.01.tar
My-Module-0.01.par
    

• Formats supported by installers and reporting tools
• .ppm, .par and non-source based distributions ignored

What to report?

• Test using a supported test mechanism
• Include results for failures
• ALWAYS include output of 'perl -V'
• Format Subject correctly

PASS My-Module-0.01 i386-freebsd 5.1-current
PASS My-Module-0.01 i686-linux 2.4.13

My-Module.tar.gz Failed
FAIL My-Module
FAIL My-Module MSWin32-x86-multi-thread 4.0
FAIL My-Module-0.01.tar.gz sun4-solaris 2.8
    

Who Is That CPAN Tester?

A Request

• Submit using a recogniseable FROM address

DAGOLDEN@cpan.org
"Barbie" <barbie@missbarbell.co.uk>

"Root" <mybox@localhost>
"Bogus" <root@a.fake.address>
    

• Can identify you in the statistical analysis
• Authors know who to respond to

Two styles of CPAN testing

• "Manual" -- during ordinary module installation
  • CPAN + CPAN::Reporter
  • CPANPLUS
• "Smoke" -- Automated, mass testing
  • CPAN::Reporter::Smoker
  • CPAN::YACSmoke
  • POE::Component::CPAN::YACSmoke

Two styles of CPAN testing

• "Manual" -- during ordinary module installation
  • CPAN + CPAN::Reporter
  • CPANPLUS
• "Smoke" -- Automated, mass testing
  • CPAN::Reporter::Smoker
  • CPAN::YACSmoke
  • POE::Component::CPAN::YACSmoke

CPAN::Reporter

CPAN::Reporter Installation

Super quick-start version...

 $ cpan

 cpan> install CPAN::Reporter

 cpan> reload CPAN

 cpan> o conf init test_report
    

CPAN::Reporter Benefits

• Adds CPAN test reporting to CPAN.pm
• Allows anyone who installs a distribution to contribute
• Allows testers to edit reports before sending
• Provides selective report submission based on configuration
• Keeps track of subject lines sent to CPAN Testers
  • Subject line has grade, module name and perl version
  • Avoids duplicate reports if re-testing

CPAN::Reporter::Smoker

CPAN::Reporter::Smoker Installation

• Install Bundle::CPAN
• Configure and test CPAN::Reporter manually first
• Quick Guide on CPAN Testers Wiki
• Detailed Guide on CPAN Testers Wiki

CPAN::Reporter::Smoker Benefits

• Turnkey based configuration
• Smokes all of CPAN, starting with most recent distributions and working backwards
• Can use CPAN.pm 'distroprefs' to skip distros that cause problems
• High-speed testing:
  • get latest CPAN.pm dev release
  • set 'trust_test_report_history', 'build_dir_reuse'

CPAN::YACSmoke

CPAN::YACSmoke Installation

• Install CPANPLUS + dependencies
• Install CPAN::YACSmoke
• Amend configuration file (if required)
• Run yacsmoke.pl
• Run yacsmoke.pl via cron or scheduler

CPAN::YACSmoke Benefits

• Multiple plugins
  • .. to retrieve what to test
  • .. to record test results locally
• Control what distributions to test
• Avoids repeat testing
• Allows retesting when a prerequisites pass
• Tools to clean up test environment & history
• Introduced $ENV{AUTOMATED_TESTING}

The CPAN Testers Community

The CPAN Testers Community

• CPAN Testers Wiki
• CPAN Testers Reports
• CPAN Testers Statistics
• CPAN Testers Matrix
• CPAN Dependencies
• cpan-tester-discuss

CPAN Testers Wiki

CPAN Testers Wiki

• http://wiki.cpantesters.org/
• Documentary resource for CPAN Testers
• About CPAN Testing
• How to get started
• Issues
• Future Plans

CPAN Testers Reports

CPAN Testers Reports

• http://wiki.cpantesters.org/
• All the distributions ever tested are recorded
• New interface highlights PASS/FAIL ratio for each version
• Soon to receive community links updates

CPAN Testers Statistics

CPAN Testers Statistics

• http://stats.cpantesters.org/
• Statistical analysis of all uploads and reports
• Highlights environments lacking CPAN Testers support
• Emphasises CPAN Testers contributions

CPAN Testers Statistics

• Last 12 months
  • 100+ regular testers
  • 60+ different platforms
  • 20+ different perls
• Last month (May 2008)
  • 1476 uploaded distributions
  • 176,813 reports submitted in total
  • 128,228 reports submitted by top 3 testers
  • 39,397 reports submitted for i686-linux-64int
  • 76199 reports submitted for 5.10.0

CPAN Testers Statistics

• Our Top Testers
  1. Chris Williams (BINGOS)
  2. Andreas J. König (ANDK)
  3. David Cantrell (DCANTRELL)
  4. David Golden (DAGOLDEN)
  5. Slaven Rezić (SREZIC)

CPAN Testers Statistics

• Notable Recent Moments
  • January 2007 - First tester to submit over 10,000 in one month (Chris Williams)
  • May 2007 - 500,000th report submitted (by Serguei Trouchelle)
  • January 2008 - 1,000,000th report submitted (by David Cantrell)
  • May 2008 - 1,500,000th report submitted (by Andreas König)
  • May 2008 - Most reports submitted in one month (176,813)
  • May 2008 - Most reports by one tester in a month (54764 by Andreas König)

CPAN Testers Statistics

• Notable Recent Moments
  • January 2007 - First tester to submit over 10,000 in one month (Chris Williams)
  • May 2007 - 500,000th report submitted (by Serguei Trouchelle)
  • January 2008 - 1,000,000th report submitted (by David Cantrell)
  • May 2008 - 1,500,000th report submitted (by Andreas König)
  • May 2008 - Most reports submitted in one month (176,813)
  • May 2008 - Most reports by one tester in a month (54764 by Andreas König)
  • June 2008 - Most reports by one tester in a month (74000+ by David Golden)

CPAN Testers Matrix

CPAN Testers Matrix

• http://bbbike.radzeit.de/~slaven/cpantestersmatrix.cgi
• Highlights CPAN Tested distribution by platform and perl
• Find latest tested releases for a distribution
• Find latest tested releases by author

CPAN Dependencies

CPAN Dependencies

• http://deps.cpantesters.org/
• For a given distributions...
  • Lists each CPAN dependency
  • Using CPAN Testers generates pass rate for each dependency
  • Then generates a probability of installation success
  • Can restrict by platform and/or perl

CPAN Testers Discuss

CPAN Testers Discuss

• Mailing List - cpan-tester-discuss@perl.org
• Highlight issues with any CPAN Tester reports
• Highlight issues with any CPAN Tester related tool or website
• Discuss future plans and wishlists

The Future

The Future

• Add more documentation to the CPAN Testers Wiki
• Encourage more CPAN Testers to test on a wider variety of platforms/perls
• More CPAN::Testers tools on CPAN
• Integration of CPAN::Testers tools into CPAN::(YACSmoke|Reporter(::Smoker)?)

The Future

• CPAN::Metabase
  • See RJBS & XDG
  • One of the results of the Oslo QA Hackathon
• CPAN Testers over HTTP
• CPAN Author Configuration
• CPAN Testers Admin Tools

Questions?

Thanks for listening.